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jDatent application is related to concurrently filed U.S. patent application 

serial nK , titled "Method of Upscaling A Color Image," by 

Acharya,(atWiey docket no. 042390. P7489), assigned to the assignee of the 
current inventiorv^nd herein incorporated by reference. 
BACKGROUND 

This disclosure relates to color conversion and, more particularly, to 
converting from a sub-sampled color image in a first color space format to a full 
color image in a second color space format. 

As is well-known, in a variety of circumstances, it is a desirable to apply a 
technique to convert from a sub-sampled color image to a full color image. For 
example, for a camera or other imager that has the capability of creating a color 
image, typically it is too expensive to have the capability to capture three 
independent color pixel signal values for each pixel location of an image of a 
particular size or dimension. Therefore, more typically, a sub-sampled color 
image is captured and the missing color pixel signal values at each pixel location 
are computed using a variety of techniques. In other words, each pixel location 
may have a single color signal value rather than three independent color signal 
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values, for example. A need continues to exist for techniques for creating a full 
color image of good quality from a sub-sampled color image. 
BRIEF DESCRIPTION OF DRAWINGS 

The subject matter regarded as the invention is particularly pointed out 
and distinctly claimed in the concluding portion of the specification. The 
invention, however, both as to organization and method of operation, together 
with objects, features, and advantages thereof, may best be understood by 
reference to the following detailed description when read with the accompanying 
drawings in w^h^ 

Fig.^/is a schematic diagram illustrating a sub-sampled color image, such 
as a Bayer pattern color image; 

Fig. 2^s a schematic diagram illustrating one technique for producing a full 
color image from^'a sub-sampled color image and subsequently converting to 
another color space format; 

Fig.yG is a schematic diagram illustrating a technique of converting from a 
sub-sampled color/image to a full color image in accordance with the present 




invention; and 

Fig. 4ns an alternative way to represent the sub-sampled color image of 

Fig. 1. 

DETAILED DESCRIPTION 

In the following detailed description, numerous specific details are set 
forth in order to provide a thorough understanding of the invention. However, it 
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will be understood by those skilled in the art that the present invention may be 
practiced without these specific details. In other instances, well-known methods, 
procedures, components, and circuits have not been described in detail so as 
not to obscure the present invention. 

As previously indicated, for a color image, typically each pixel signal value 
is represented by three independent color components, such as a red 
component, a green component, and a blue component, for example. However, 
the invention is not limited in scope to these color components or to this 
particular color space format. Ideally, three image sensors would be used to 
capture the three color components. However, the use of three sensors in a 
single image capture device is typically expensive and also may not be suitable 
for use in low power handheld devices, such as a mobile handset designed to 
display such color images, for example. 

In order to reduce cost, typically the color image is captured in a sub- 
sampled form using a single sensor in each pixel location with a suitable color 
filter array (CFA) "on top" of the sensor. Of course, the invention is not limited in 
scope to use with such an image capture device, however. Nonetheless, in such 
an approach, each pixel location comprises only one color component signal 
value, thereby forming a type of sub-sampled color image. For example, in one 
approach, each pixel signal value may comprise either a red color signal value, a 
green color signal value, or a blue color signal value represented by eight binary 
digital signals or bits. Although the invention is not limited in scope in this 
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respect, one such pattern is a Bayer pattern, as illustrated schematically in Fig, 
1. 

Of course, for a color image formed using three color sensors in each 
pixel location, each location would include signal information for all three 
independent color components. Therefore, typically, for a color image captured 
by the latter device, each pixel is represented by a 24 bit color signal, 8 bits for 
each of the red, green, and blue color components, for example. Again, the 
invention is not restricted in scope by this example; however, in order to produce 
such an image from a device employing one color sensor in each pixel location, 
for example, signal interpolation would typically be employed. In one example of 
signal interpolation, the two missing color components for each pixel location are 
obtained by using color signal information provided by neighboring pixel signal 
values. 

If a full color image is obtained in this manner, a transformation into a 
different color space format may then be employed. For example, it may be 
desirable to comply with certain image processing specifications and/or 
standards, such as those provided by the International Telecommunications 
Union (ITU) or the International Standards Organization (ISO), although the 
invention is not limited in scope in this respect. Likewise, it is also typical to 
represent color images in a luminance-chrominance (L-C) form in order to 
reduce the storage requirements by sub-sampling redundant color signal 
information that may be present in the image, typically in the chrominance color 
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components. One widely used L-C color space format is the Y-Cr-Cb 4:2:0 
representation, although, again, this is just an example. 

Disadvantages of current techniques to produce a full color image from a 
sub-sampled color image include (1) the loss of signal information as a result of 
applying the particular technique and/or (2) high computational complexity, which 
also may typically consume excessive power when employed or performed by a 
computing engine, and/or (3) internal buffering large enough to store the signal 
information during and after processing. These disadvantages will be explained 
in more detail hereinafter. An embodiment of converting from a sub-sampled 
color image to a full color image in accordance with the present invention may 
avoid these disadvantages. 

As is well-known, a color interpolation method or process, such as 
described in "Method of Interpolating Color Pixel Signals from a Subsampled 
Color Image," by Acharya et al., filed on October 1, 1999, US patent application 
serial number 09/41 0,800,(attorney docket no. 042390. P7331), assigned to the 
assignee of the present invention, for example, generates a full color digital 
image comprising three color planes, red, green, and blue, each of size M x N 
from a Bayer pattern sub-sampled image by generating the missing two color 
components for each pixel location. Of course, this is just one example of a 
color interpolation process, and the invention is not limited in scope to this or any 
other particular approach. There are a wide number of color interpolation 
techniques that may be employed. Typically, the color signal information of the 
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neighboring pixels are employed to generate the missing color signal information 
for each pixel. A simple approach to implement would be to employ the median 
of the neighboring pixels. Of course, more complex approaches may also be 
employed. Fig. 2 illustrates a process in which a sub-sampled image is 
interpolated to produce a full color image. 

Subsequently, for the approach illustrated in Fig. 2, an interpolated full 
color image is converted to a sub-sampled luminance-chrominance (L-C) color 
space format, such as Y-Cr-Cb 4:2:0, to meet color specifications or criteria, 
such as provided, for example, in the H.263 ITU standard, applied in video 
conferencing applications. Of course, the invention is not restricted in scope in 
this respect. The color space format conversion from 24 bit RGB color space 
format to 12 bit Y-Cr-Cb 4:2:0 color space format is done using the following 
transformation: 



Cb(l,J) = 0.500R(I,J) - 0.419G(I,J) - 0.081 B(I,J) 

where R,G, and B denote different color planes in the RGB color space 
format, Y, Or, and Ob denote different color planes in the Y-Cr-Cb 4:2:0 color 
space format, and I, J denote pixel locations. 



Y(I,J) = 0.299R(I,J) + 0.587G(I,J) + 0.114B(I,J) 



Cr(l,J) = -0.169R(I,J) - 0.331 G(I,J) + 0.500B(I,J) 



[1] 
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The Cr and Cb image planes are then, in this particular approach, 
decimated by dropping alternate rows and columns from each chrominance 
plane (Cb and Cr), as illustrated by Cr' or Cb' in Fig. 2. This generates the Y-Cr- 
Cb image in 4:2:0 color space (12 bit color) format. The reason for 12 bits for 
each pixel in Y-Cr-Cb 4:2:0 color space is that for every 4 luminance (Y) pixels (1 
byte each for each luminance value) in this format there is 1 value (1 byte each) 
for each corresponding chrominance channel (Cb and Cr in this example). 
Hence, 4 full color pixels in this format are represented by 6 bytes or 48 bits. As 
a result, each pixel in Y-Cr-Cb color space format is considered to have 12 bits. 

Assume that the average number of operations to recover each missing 
color signal value in a particular pixel location is "K", when using a known color 
interpolation technique. Then, to recover two missing color space components 
for each pixel location, the total number of operations to restore the full color 
image using this approach is 2Kx(MxN), where "x" denotes multiplication. The 
total number of multiplications for color conversion is 9MxN, and the number of 
additions is 6MxN. Therefore, ignoring the computation for sub-sampling, the 
typical number of operations for color interpolation followed by color conversion 
is more than (2K + 15)xMxN. Likewise, the memory buffer size employed for 
hardware and/or software implementation of this approach is 3MxN. 

Fig. 3 illustrates an embodiment in accordance with the present invention. 
Of course, the invention is not limited in scope to this particular embodiment. In 
this embodiment, however, a sub-sampled color image in a first color space 



8 




42390-P7490 




15 



20 



format, such as a Bayer pattern in RGB color space, is transformed to a second 
color space format, such as Y-Cr-Cb color space format. In this particular 
embodiment, the Bayer pattern of size M x N is converted to an image of size 
M/2 X N/2 using an integrated approach that avoids computationally complex 
color interpolation methodologies and the decimation operation previously 
described. Then, at least one of the color space dimensions in the transformed 
image is "upscaled" to provide a full color image in a second color space format. 

In this context, the term upscaled refers to a process applied to a color 
plane of a color dimension in a particular color space format whereby the color 
signal information in the color plane is retained, but spread or distributed over a 
color plane of larger dimensions. For example, for the embodiment illustrated in 
Fig 3, the Y plane of size M/2 x N/2 is upscaled using a discrete wavelet 
transform (DWT) based upsampling technique in order to generate an M x N 
Y-Cb-Cr 4:2:0 color image. Of course, the invention is not limited in scope to 
using the discrete wavelet transform and many other sampling techniques may 
be employed such as bi-linear interpolation, weighted averaging,etc. An 
advantage of this particular approach is that the chrominance plane is not 
decimated and, hence, there is no significant loss of signal information as would 
typically occur. 

To illustrate this particular embodiment in greater detail, Fig. 4 is a 
schematic diagram that represents the Bayer pattern of Fig. 1 differently than in 
Fig. 1. Each block B(I,J) represents a 2 x 2 Bayer pattern as shown in Fig. 4. 
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Integrated color interpolation and color space conversion is accomplished using 
the previous transfornnation or equations [1] with the following additional 
relationships applying also: 
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R(I,J) = R 

G(I,J) = (G1 +G2)/2 [2] 
B(I,J) = B 

where R, G1 , G2, and B are from the block B(I,J). 



Ko As a result of this transformation, a Y-Cr-Cb color Image of size M/2 x N/2 Is 



obtained because each 2x2 Block produces a single pixel signal value in this 
color space format. The Y plane may then be upsampled to generate the M x N 
1=4 size Y-Cr-Cb 4:2:2 color space format image. One technique for performing this 

upscaling is described in greater detail in a prior patent application "A DWT- 
15 based Up-Sampling Algorithm Suitable For Image Display In An LCD panel," by 



/ , Acharya, filed on August 5, 1998, US patent application serial no. 



09/129,728,(Docket No. 042390.P6080), assigned to the assignee of the present 



invention, although the invention is not limited in scope in this respect.. As 
previously indicated any one of a number of techniques to perform the upscaling 
20 may be employed. 

It is noted that the computational complexity of this approach and the 
amount of storage space utilized is significantly less than the previous approach. 
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As discussed above, the total number of multiplications for an integrated color 
interpolation and color conversion is 9x(M/2)x(N/2) or (9/4)MxN and a similar 
number of additions and subtractions. Hence, the total number of operations is 
4,5 MxN. In this particular embodiment, for an upscaling operation for the Y 
plane, the computational complexity is similar to that of color interpolation; 
however, the upscaling is applied to data of size M/2 x N/2. Hence, for upscaling 
the Y plane, the total number of operations is KxMxN. Hence, the total number 
of operations for this particular embodiment is (K+4.5)xMxN, which is less than 
half of the approach previously described in terms of computational complexity. 
Likewise, the storage space employed to implement this approach is 1 .5MxN or 
about half of the approach previously described. 

As previously indicated, an embodiment of a method of converting from a 
sub-sampled color image in a first color space format to a full color image in a 
second color space format in accordance with the invention provides a number 
of advantages. The color interpolation approach is integrated with the color 
conversion methodology. As a result, a computationally complex color 
interpolation scheme is avoided totally with the trade off being the addition of an 
upscaling scheme applied to a reduced set of data. Because of this integrated 
approach, pipelining the imaging processing flow, as is desirable in implementing 
video, for example, is relative straightfonward and useful for such applications. 
The reduction in computational complexity is more than 50%, and in memory 
storage space is about 50%. Therefore, this approach may be applied to 
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virtually any digital signal processing architecture or similar computing engine 
and may reduce computational complexities suitable for real-time application and 
low power applications, such as may be desirable for handheld devices, with 
generally no significant loss of signal information. 

It will, of course, be understood that, although a particular embodiment 
has just been described, the invention is not limited in scope to a particular 
embodiment or implementation. For example, one embodiment may be in 
hardware, whereas another embodiment may be in software. Likewise, an 
embodiment may be in firmware, or any combination of hardware, software, or 
firmware, for example. Likewise, although the invention is not limited in scope in 
this respect, one embodiment may comprise an article, such as a storage 
medium. Such a storage medium, such as, for example, a CD-ROM, or a disk, 
may have stored thereon instructions, which when executed by a system, such 
as a computer system or platform, or an imaging system, may result in a method 
of upscaling a color image in accordance with the invention, such as, for 
example, the embodiment previously described. 

While certain features of the invention have been illustrated and described 
herein, many modifications, substitutions, changes and equivalents will now 
occur to those skilled in the art. It is, therefore, to be understood that the 
appended claims are intended to cover all such modifications and changes as 
fall within the true spirit of the invention. 
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